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Abstract 

Background: Physicians practicing in ambulatory care are adopting electronic health record (EHR) systems. 
Governments promote this adoption with financial incentives, some hinged on improvements in care. These 
systems can improve care but most demonstrations of successful systems come from a few highly computerized 
academic environments. Those findings may not be generalizable to typical ambulatory settings, where evidence of 
success is largely anecdotal, with little or no use of rigorous methods. The purpose of our pilot study was to 
evaluate the impact of a diabetes specific chronic disease management system (CDMS) on recording of information 
pertinent to guideline-concordant diabetes care and to plan for larger, more conclusive studies. 

Methods: Using a before-after study design we analyzed the medical record of approximately 10 patients from 
each of 3 diabetes specialists (total = 31) who were seen both before and after the implementation of a CDMS. We 
used a checklist of key clinical data to compare the completeness of information recorded in the CDMS record to 
both the clinical note sent to the primary care physician based on that same encounter and the clinical note sent 
to the primary care physician based on the visit that occurred prior to the implementation of the CDMS, 
accounting for provider effects with Generalized Estimating Equations. 

Results: The CDMS record outperformed by a substantial margin dictated notes created for the same encounter. 
Only 10.1% (95% CI, 7.7% to 12.3%) of the clinically important data were missing from the CDMS chart compared to 
25.8% (95% CI, 20.5% to 31.1%) from the clinical note prepared at the time (p < 0.001) and 26.3% (95% CI, 19.5% to 
33.0%) from the clinical note prepared before the CDMS was implemented [p< 0.001). There was no significant 
difference between dictated notes created for the CDMS-assisted encounter and those created for usual care 
encounters (absolute mean difference, 0.8%; 95% CI, -8.5% to 6.8%). 

Conclusions: The CDMS chart captured information important for the management of diabetes more often than 
dictated notes created with or without its use but we were unable to detect a difference in completeness between 
notes dictated in CDMS-associated and usual-care encounters. Our sample of patients and providers was small, and 
completeness of records may not reflect quality of care. 



Background 

Successful diabetes care requires the active participation 
and informed self-management of engaged, educated, and 
motivated patients. The healthcare team, in turn, requires 
timely access at the point of care to comprehensive patient 
information, including but not limited to weight, blood 
pressure, current medications, glycosylated haemoglobin 

* Correspondence: bhaynes@mcmaster.ca 

department of Medicine, McMaster University, 1280 Main Street West, 
Hamilton, ON, Canada 

Hamilton Health Sciences, 1200 Main Street West, Hamilton, ON, Canada 
Full list of author information is available at the end of the article 

(3 BioMed Central 



(HbAlc), serum creatinine, LDL cholesterol, and urine al- 
bumin to creatinine ratio. Easy access to this information 
helps clinicians and patients to tailor and optimize care. 
Collection of standardized and well-defined data on each 
patient also allows for provider-level and clinic-level sum- 
mary data for quality improvement. 

An electronic, diabetes-specific, chronic disease man- 
agement system (CDMS) that collects limited data dur- 
ing the clinical encounter and presents these data during 
future clinical encounters may help meet the needs of 
diabetic patients and their healthcare team. 
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In a series of 6 systematic reviews covering 166 rando- 
mized controlled trials (RCTs), we recently synthesized 
the evidence of effectiveness of computerized clinical de- 
cision support systems for primary prevention [1], diag- 
nostic test ordering [2], acute care [3], drug prescribing 
and management [4], therapeutic drug monitoring and 
dosing [5], and management of chronic diseases [6]. Sev- 
eral systems for management of diabetes have been 
tested in RCTs and a small majority improve practitioner 
performance, with less success for enhancing patient 
outcomes [7-25]. 

However, creating effective clinical information systems 
remains difficult. Because most systems are developed and 
tested in a few highly computerized environments with a 
long history of informatics excellence, the findings may 
not be generalizable to the more technologically naive set- 
tings where most people receive care. Recent attention on 
health information technology as a means for better 
healthcare has made it increasingly important to design 
and test systems in a variety of settings. 

The P-PROMPT CDMS in the Diabetes Care and Research 
Centre 

With the help of diabetes specialists, Fig.P Software Inc. 
tailored its web-based chronic disease management sys- 
tem, the P-PROMPT CDMS, specifically for diabetes 
care and deployed it in the Hamilton Health Sciences 
Diabetes Care and Research Centre (DCRC), a hospital- 
based, ambulatory care subspecialty clinic. The systems 
record structure is based on clinical practice in the 
DCRC and on the Canadian Diabetes Association prac- 
tice guidelines for 2008, updated by us to 2010 to re- 
flect more recent evidence. In a typical patient visit to 
the DCRC, physicians interview and examine patients 
in a room with an internet-connected computer. They 
always dictate a note summarizing the patient visit. 
Some practitioners tend to dictate their summary im- 
mediately after the patient leaves, while others defer 
this until the end of the day. Once transcribed, this dic- 
tated note is stored in the patients paper records and 
communicated to their family doctor. Prior to the 
DCRCs recent adoption of the CDMS, this note, and 
any laboratory test results attached to it, comprised the 
longitudinal record of each patient. 

Figures 1, 2, 3 show screenshots of the CDMS inter- 
face. During or after a patient visit, physicians enter dir- 
ectly into the CDMS information important for the 
management of diabetes, including each patients demo- 
graphic information, existing diagnoses, disease registry 
membership, and complete medication profile that 
includes current and past medications, along with rea- 
sons for discontinuation. The system is linked to the 
hospitals electronic laboratory records and summarizes 
the results of laboratory tests in a dashboard that 



displays longitudinal trends graphically; results from 
tests performed outside of the hospital must be entered 
manually. Finally, the CDMS can provide point-of-care, 
guideline-based, patient-specific decision support for 
practitioners and tailored self-management support mes- 
sages intended for patients. 

Before carrying out a large study to test the impact of 
the CDMS on diabetes care in the DCRC, we conducted 
a small pilot to help with sample size calculations. For 
this study we asked, "Does the CDMS impact the record- 
ing of clinical information pertinent to evidence-based 
diabetes care?" 

Methods 

Figure 4 summarizes the study design. This was a retro- 
spective chart review of recent patient record entries 
(electronic charts and transcripts of dictated notes with 
any attached laboratory reports) generated as a result of 
CDMS -associated visits and previous transcripts of dic- 
tated notes (and attached laboratory reports) created 
without the help of the CDMS for the same patients. 
The Hamilton Health Sciences Research Ethics Board 
provided ethics approval for this study. 

Eligibility of patient records 

Patient records were eligible for review for patients who 
had their first complete CDMS -associated assessment at 
the clinic between January 1st 2011 and April 1st 2011 
and previously without the help of the CDMS between 
June 1st 2009 and June 1st 2010, with no intervening 
DCRC physician visit (ignoring visits made to other 
health professionals). 

Primary outcome 

16 items of particular interest included the patients 
weight; blood pressure; use of statins when appropriate; 
use of Angiotensin Converting Enzyme (ACE) inhibitors 
or Angiotensin II Receptor Blockers (ARBs) if appropri- 
ate; HbAlc; serum creatinine; LDL cholesterol; urine al- 
bumin to creatinine ratio (UACR); eye and foot 
examinations within the past year; any occurrence of se- 
vere hypoglycaemic episodes; number of non-severe 
hypoglycaemic episodes per month; cardiovascular 
events since the previous visit; and perfusion, light touch 
sensation, and skin integrity of the feet. 

We counted the number of items in each record and 
calculated a missing fraction using Formula 1 

, . . f 16 — present\ , N 

Missing Fraction = ( ^ J x 100 W 

For example, a record with 12 present items would 
have a missing fraction of 25%. We found the mean 
missing fraction by first finding the missing fraction 
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Figure 2 CDMS diabetes checklist. 
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Figure 3 CDMS medications. 

k J 



for each patients records and calculating their mean. 
We considered items to be present if clearly stated in 
the record or not relevant for a given patient. We 
used values for blood pressure and UACR to judge 
whether a patients use of an ACE inhibitor or ARB 
should have been noted in that patients record, as 
the relevance of these medications depends on those 
values. ACE inhibitors or ARBs are appropriate if the 
UACR is greater than 2.0, or if the patient is known 
to be hypertensive or blood pressure measured in 
clinic is above 130/80 mmHg. Statin medications are 
appropriate if LDL cholesterol levels exceed 2.0 mmol/L, 
if a diabetic patient is male and over 45 years of age or fe- 
male and over 50 years of age, or if the patient has ever 
experienced a stroke, myocardial infarction, or other 
major vascular event. 

Our outcome of interest was the mean absolute differ- 
ence in missing fraction between pairs of record types 
(see Analysis). Note that the CDMS does not directly as- 
sist dictation. CDMS- associated dictation refers to dic- 
tated notes that correspond to patient visits in which the 
CDMS was used. 



Measurement protocol 

From a larger group of 7 physicians in the DCRC, we 
selected for participation in this pilot study 3 attending 
physicians with a special interest in medical informatics. 
First, each clinician screened his patients' records sequen- 
tially for inclusion according to the eligibility criteria, be- 
ginning with the record updated most recently and 
moving to records updated progressively earlier until 10 
eligible patient records were identified. Second, documen- 
tation pertaining to 2 separate clinical encounters was 
assessed within each record: the documentation generated 
during the recent CDMS -associated encounter and that 
which was generated during the next most recent encoun- 
ter that did not include the CDMS, for a total of 61 
encounters documented in 31 patient records. a 

Records were assessed in two phases: 1) review of all 
dictated notes and 2) review of electronic charts from 
the CDMS (depending on whether the CDMS was 
used or not) created during the corresponding patient 
encounter. We ordered these review phases to minimize 
any impact of the CDMS chart assessment on assess- 
ment of the dictated note. 
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Patient Record Eligibility Criteria 

1 . A record from a repeat visitor to the Diabetes Care and Research Program 
ambulatory clinic assessed with the help of the diabetes CDMS between Jan 1st 2011 
and April 1st 2011. 

2. Record should have second most recent entry created between June 1st 2009 and 
June 1st 2010 in a usual care (non-CDMS) visit. 



1. Sequentially identify 10 eligible patient records per provider 
beginning with those that contain most recent entries 

IN EACH RECORD: 

3. Abstract data from 2. Abstract data from 

entries generated in a past ^ entries generated in a 

usual care encounter. CDMS-assisted encounter. 




June 1 st 2009 and June 1 st 201 0 



Jan 1st 2011 and April 1st 2011 




if missing ACEi/ARB or statin and 
no corresponding value for 
BP, LDL-C, or UACR present, 
then check 
Meditech and office chart 
for LDL-C or UACR 



if missing ACEi/ARB or statin and 
no corresponding value for 
BP, LDL-C, or UACR present, 
then check 
Meditech and office chart 
for LDL-C or UACR 



Comparison 1 : CDMS chart + CDMS-assisted transcript of dictation VS usual care transcript of 
dictation. 



Comparison 2: CDMS-associated transcript of dictation VS usual care transcript of dictation. 

Outcome: Number of the following diabetes metrics recorded: weight, blood pressure, use of ACE 
inhibitor (if appropriate), use of statin (if appropriate), HbA1c, serum creatinine, LDL cholesterol, urine 
albumin to creatinine ratio, eye examination within the past year, foot examination within the past year, 
severe hypoglycemic episodes, non-severe hypoglycemic episodes per month, CV event since last visit, 
perfusion of feet, light touch sensation in feet, and integrity of skin on feet. 



Figure 4 Study design. CDMS, Chronic disease management system; BP, Blood pressure; ACEi, Angiotensin Converting Enzyme inhibitor; ARB, 
Angiotensin II Receptor Blocker; HbA1C, Glycosylated haemoglobin; Cr, Creatinine; LDL, Low density lipoprotein cholesterol; UACR, Urinary 
albumin to creatinine ratio; CV, Cardiovascular. 



Analysis 

We estimated the mean differences and standard devia- 
tions between patient records for the proportion of 
missing items. We assessed 3 different record types for 
each patient (transcript of dictation from visit in which 
the CDMS was used [CDMS-associated transcript], tran- 
script of dictation from usual care [non-CDMS] visit, 
and CDMS chart). For each patient, we calculated the 
absolute difference in the missing fraction in each of the 
following pairs: 

1. CDMS-associated transcript of dictation VS usual 
care transcript of dictation 

2. CDMS chart VS CDMS-associated transcript of 
dictation 

3. CDMS chart VS usual care transcript of dictation 

All patients received care from 1 of 3 providers. We 
used Generalized Estimating Equations (GEE) [26] to 



perform our comparisons. GEE allowed us to construct 
multi-level linear models with a structure that takes into 
account similarity between records created by the same 
provider. We measured this similarity using the Intra- 
class Correlation Coefficient (ICC) [26]. The ICC ranges 
from 0 to 1; values close to 0 mean that there is little re- 
lationship between the records of the same provider 
while values closer to 1 suggest a strong relationship. An 
ICC of 1 indicates that completeness for a given type of 
record is constant for records created by the same pro- 
vider, making the effective sample size of the study 3 in- 
stead of 31. We performed all analyses using the STATA 
Statistical Software package, version 11.2 [27]. 

Results 

Table 1 describes the data; Table 2 presents results of 
the statistical comparisons and, along with Figure 5, 
summarizes missing fractions corresponding to each 
record type overall and by provider. 
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Table 1 Descriptive statistics 



Group 


N 


"Missing" fraction 








Mean (SEM)% 


95% CI 


Usual-care note 


30 


26.25 (3.32) 


19.47, 33.03 


Provider 1 


10 


23.75 (5.00) 


12.44, 35.06 


Provider 2 


9 


43.75 (5.89) 


30.16, 57.34 


Provider 3 


1 1 


14.20 (1.90) 


9.97, 18.44 


CDMS-associated note 


"3 1 

3 l 


ZD.O 1 (Z.OUJ 


on a o 

zU.4o, Sl.lZ 


Provider 1 


10 


21.25 (2.98) 


14.52, 27.98 


Provider 2 


10 


38.13 (4.00) 


29.07, 47.18 


Provider 3 


11 


18.75 (4.04) 


9.74, 27.76 


CDMS chart 


31 


10.08 (1.18) 


7.66, 12.50 


Provider 1 


10 


10.00 (1.02) 


7.69, 12.31 


Provider 2 


10 


15.00 (1.38) 


11.87, 18.13 


Provider 3 


11 


5.68 (2.30) 


0.55, 10.81 



CDMS, Chronic disease management system; SEM, Standard error of the mean; 
CI, Confidence interval. 



The CDMS chart outperformed the CDMS-associated 
dictated note by a substantial margin (mean missing 
fraction, 10.1%, 95% CI [7.7% to 12.3%] vs. 25.8%, 95% 
CI [20.5% to 31.1%], respectively; absolute difference, 
15.8%, 95% CI [11.4% to 20.1%]). It similarly outper- 
formed the usual-care dictated note (mean missing frac- 
tion, 26.3%, 95% CI [19.5% to 33.0%]; absolute mean 
difference, 16.5%, 95% CI [10.6% to 22.4%]). However, 
there was no difference between the usual-care dictated 
note and the CDMS-associated dictated note (absolute 
mean difference, 0.8%; 95% CI, -8.5% to 6.8%). 

Figure 6 presents the number of times each of the 16 
items was considered missing across the 3 record types. 



Table 2 Comparison results 



Group 


N 


Absolute difference in"Missing" fraction 






Mean A (SEM*)% 


95% CP 


CDMS-associated 
note - CDMS chart 


31 


15.75 (2.22) 


11.39, 20.11** 


Provider 1 


10 


1 1 .25 (3.20) 


4.01, 18.49** 


Provider 2 


10 


23.13 (3.85) 


14.42, 31.83** 


Provider 3 


11 


13.07 (4.08) 


3.97, 22.16** 


Usual-care - 
CDMS chart 


30 


16.47 (3.02) 


10.55, 22.41** 


Provider 1 


10 


13.75 (5.65) 


0.96, 26.54** 


Provider 2 


9 


29.17 (6.25) 


14.75, 43.58** 


Provider 3 


11 


8.52 (3.29) 


1.18, 15.86** 


Usual-care note - 
CDMS-associated 


30 


0.82% (3.91) 


-8.48, 6.84 


Provider 1 


10 


2.50 (4.95) 


-8.69, 13.69 


Provider 2 


9 


5.56 (6.46) 


-9.34, 20.45 


Provider 3 


11 


-4.55 (4.62) 


-14.85,5.76 



*Estimated assuming an exchangeable covariance structure [26]; **p<0.05; 
CDMS, Chronic disease management system; SEM, Standard error of the mean; 
CI, Confidence interval. 



We did not conduct statistical tests to determine the 
greatest contributors to the missing fraction, but serum 
creatinine, UACR, hypoglycemia, and perfusion of feet 
appear to contribute most often. 

The ICC for the difference in missing fraction between 
CDMS charts and CDMS-associated dictation tran- 
scripts created by the same practitioner was 0.15 (95 CI, 
0.00 to 0.55), 0.25 (0.00 to 0.75) for the difference in 
missing fraction between CDMS charts and usual care 
transcripts, and 0 (0.00 to 0.20) for the difference in miss- 
ing fraction between CDMS-associated transcript and 
usual care transcript. These estimates are very imprecise 
because only three providers were included in the study. 

Discussion 

We discovered that CDMS charts capture information 
important for the management of diabetes more often 
than dictated notes created with or without its use. 
There was no difference between the two types of dic- 
tated notes, suggesting that use of the CDMS neither 
improved nor hindered the quality of the dictated note 
among diabetes specialists. We chose the target sample 
size of 30 patients to help us calculate sample size for fu- 
ture studies of this system, and did not explicitly power 
the study to detect differences in completeness. With so 
few patients, we cannot dismiss the possibility of positive 
or negative impact. Indeed, confidence intervals ranged 
from substantial improvement to substantial decline in 
completeness of dictated notes with use of the CDMS. 

Completeness of records is only a distant surrogate 
marker of clinical care quality. It is possible that the 
items in our assessment form were biased toward the 
CDMS chart. The CDMS may elicit responses to items 
that the practitioner deems irrelevant for the dictated 
summary of a particular clinical encounter. For example, 
an unremarkable foot exam may not be mentioned in 
the dictation but the practitioner may still indicate in 
the CDMS chart (which provides a relevant drop-down 
menu) that it was performed. In this situation, the pa- 
tient does not gain additional benefit despite having a 
more complete CDMS chart. 

While the dictated notes did not change with CDMS 
use, storing more complete records in a structured elec- 
tronic format allows for other quality improvement 
interventions. For example, the CDMS includes point- 
of-care, guideline-based, patient-specific decision sup- 
port for practitioners and tailored self-management 
support messages intended for patients. A number of 
systems that engage diabetic patients and their practi- 
tioners have demonstrated benefit in randomized trials 
[8-10, 12]. The CDMS is also being enhanced to provide 
periodic clinician-level performance feedback, which was 
previously found to improve the quality of care when 
combined with patient-specific decision support [28]. 
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Does the CDMS improve recording of key parameters? 
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Figure 5 Does the CDMS improve recording of key parameters? Key parameters were missing less often from the CDMS chart than from 
dictated notes. Error bars indicate 95% confidence intervals. 



Providers were happy with the ability to track patients' 
progress longitudinally, and with access to past test 
results and medication profiles. We can expect some 
benefit to patient health if particular management deci- 
sions require that the practitioner refer to a detailed his- 
torical care record. Studies much larger and lengthier 



than are typically feasible with local computerized inter- 
ventions would be necessary to detect these benefits, 
which may be rare but still meaningful given the high 
prevalence of diabetes. 

A specialized diabetes clinic appears to lend itself well 
to the use of an electronic record and disease 



Which key parameters are missed and how often? 
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Figure 6 Which key parameters are missed and how often? CDMS, Chronic disease management system; BP, Blood pressure; ACEi, 
Angiotensin Converting Enzyme inhibitor; ARB, Angiotensin II Receptor Blocker; HbA1C, Glycosylated haemoglobin; Cr, Creatinine; LDL, Low 
density lipoprotein cholesterol; UACR, Urinary albumin to creatinine ratio; CV, Cardiovascular. 
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management system tailored to the needs of diabetes 
specialists and their patients. Our findings, however, 
may not be generalizable to primary care, where most 
diabetic patients are managed. Primary care physicians 
address a wide range of ailments within a single patient 
due to multimorbidity [29] and across patients owing to 
a broad scope of practice. It may, therefore, be difficult 
for general practitioners to adopt a detailed system for 
the management of just one disease. 
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